We thank the reviewers for the insightful comments. Due to space limitation, we only discuss major comments below. This example is shown in Fig(a) below. This has been shown for ECE (e.g., Sec. 3 of [i], pointed out by To further understand this, in Sec. D.2 we evaluate the performance of all D.1 due to its adaptive binning scheme (see We will update Sec D.1 as follows: Before giving the fooling example, we highlight that ECE is not a proper We were not able to finish the OOD experiments on time and have to do it in future work.

We thank the reviewers for the insightful comments. Due to space limitation, we only discuss major comments below. This example is shown in Fig(a) below. This has been shown for ECE (e.g., Sec. 3 of [i], pointed out by To further understand this, in Sec. D.2 we evaluate the performance of all D.1 due to its adaptive binning scheme (see We will update Sec D.1 as follows: Before giving the fooling example, we highlight that ECE is not a proper We were not able to finish the OOD experiments on time and have to do it in future work.

Models leveraging both visual and textual data such as Contrastive Language-Image Pre-training (CLIP), are increasingly gaining importance. In this work, we show that despite their versatility, such models are vulnerable to what we refer to as fooling master images. Fooling master images are capable of maximizing the confidence score of a CLIP model for a significant number of widely varying prompts, while being unrecognizable for humans. We demonstrate how fooling master images can be mined by searching the latent space of generative models by means of an evolution strategy or stochastic gradient descent. We investigate the properties of the mined fooling master images, and find that images trained on a small number of image captions potentially generalize to a much larger number of semantically related captions. Further, we evaluate two possible mitigation strategies and find that vulnerability to fooling master examples is closely related to a modality gap in contrastive pre-trained multi-modal networks. From the perspective of vulnerability to off-manifold attacks, we therefore argue for the mitigation of modality gaps in CLIP and related multi-modal approaches. Source code and mined CLIPMasterPrints are available at https://github.com/matfrei/CLIPMasterPrints.

Deep neural networks have achieved impressive performance and become de-facto standard in many tasks. However, phenomena such as adversarial examples and fooling examples hint that the generalization they make is flawed. We argue that the problem roots in their distributed and connected nature and propose remedies inspired by propositional logic. Our experiments show that the proposed models are more local and better at resisting fooling and adversarial examples. By means of an ablation analysis, we reveal insights into adversarial examples and suggest a new hypothesis on their origins.